Sony's controller tech is literally bananas | Patent Trawling

Games industry patents are an interesting mix of futuristic ideas, practical applications, and utter nonsense. They can tip a company's hand as to what it's planning next, give an idea of how they're thinking about certain parts of their business, or let us revisit old successes and see how they were originally envisioned. Patent Trawling is intended to be a mix of all of the above. Not everything featured here has seen a commercial release, or ever will, but it will hopefully offer some insight into why the industry is the way it is, and where it might be headed next.

Forget PlayStation, we want a PlantainStation

Game controllers have too much technology in them these days. Wireless connections, analog sticks, d-pads, shoulder and face buttons, touch pads, microphones, speakers, haptic feedback. In the end, modern peripherals are often complicated, expensive, and a hassle to keep charged

Sony apparently wants to do something about that, according to a recently published patent application.

"It would be desirable if a user could use an inexpensive, simple and non-electronic device as a video game peripheral," the application reads. "The present disclosure seeks to address or at least alleviate some of the above-identified problems."

As you might have guessed from the headline, Sony's answer to the problem is bananas. Well, not just bananas. What the application describes is a method that works with any "non-luminous passive object being held by a user." It could be a mug, a pen, a glass, or as in the inventors' preferred examples, bananas and oranges.

A camera gets images of the items in users' hands, and tracks the items based on pixels, contours, and/or colors in the images rather than QR codes or other such tracking techniques. A game could either be trained to recognize objects as controllers, or pre-configured such that users are told ahead of time what could be used as a controller. The application doesn't mention it specifically, but this latter application sounds like a more cost-effective way to approximate toys-to-life functionality in games.

The application goes on to discuss ways to infer the banana's movement in 3D space, which might be used to control the in-game camera, replace a flight stick, or pause the game if the banana were to be set down out of the camera's view.

There's also a nice section on a "Two-Object Controller," which instantly translates to us as, "Yes, you can dual-wield bananas."

The patent application also raises the possibility of holding two oranges and pretending they're a steering wheel, or using them for Battlezone-style tank controls.

Of course, sometimes motion controls aren't going to get the job done. And that's when this patent application proposes mapping virtual buttons onto the banana, possibly with the help of a camera mounted in a virtual reality headset.

Unfortunately, there's no telling if Sony's patent application will ever bear fruit -- or its players, for that matter -- so let's take a look at some patents more closely related to a product than produce.

More PSVR 2 possibilities

Last time in this column, we looked at some of the more fanciful VR tech Sony has tried to patent. Much like the banana controllers above, VR headsets that blow hot air and spray different odors in the player's face are fun advances that we aren't necessarily expecting Sony to roll out commercially.

But this one is a little different. This application for a "Control apparatus and control program" was published in November and contains a detailed description of what might be the new PlayStation VR controller Sony announced to go with the new VR headset last week.

A pair of drawings in the patent show off the controller from the front and back. The most instantly noticeable change is that the giant light-up globe of the original PSVR / PlayStation Move controller is gone, replaced by a small LED panel (labelled 235 in the drawing above) that can emit light in a variety of colors. Naturally, it has a few interesting additions as well.

First and foremost among those additions is a trigger button that, like the PS5 DualSense triggers, can offer resistance when pressed, with a vibration device to provide accompanying haptic feedback. This isn't terribly surprising, given that Sony has already said its new VR controller would incorporate technology from the PS5 pads.

However, this controller also sports a thumb sensor and three different finger sensors, so finger tracking somewhat like the Valve Index may be in play. Sony's patent application suggests that, combined with the haptic feedback, this could be used to let players feel the size and texture of virtual objects as they hold them.

And while the controller specifically detailed in the patent only includes this force feedback on the index finger button, Sony specifies that multiple buttons could be arranged to provide differing levels of feedback for each finger. The described controller also features a flexible loop that goes around the user's hand to secure the controller in place and can be mounted on either side to accommodate use in the left or right hand.

This isn't the only Sony patent featuring a controller like this. In fact, Sony has been showing versions of this controller in its patent applications since 2016, though from what we've been able to find, they didn't start appearing in publicly published applications until early 2019. There have also been reports of recent Sony patents showing other form factors for VR controllers, although we had trouble locating the source patents ourselves.

Regardless of what the new controllers actually look like or what features they support, Sony is continuing to explore ideas for finger tracking in VR. In a separate patent application filed in 2018 and published just last week, Sony details a few ideas it has for finger-based input.

In one example, the company envisions a painting program where players hold a virtual palette in one hand and use their fingers to dab the color of paint they want and draw marks across a canvas. Each finger could be assigned a different color, or be used to erase previous work.

Another example might lend itself to a sculpting program, with users deforming a 3D object, assigning different fingers to bulge, dent, shave, or smooth the object as they see fit.

As for applications that would be more at home in a game than standard art software, Sony suggested users burning or freezing virtual objects by touching them, causing something to emerge from the object where it's touched, or restoring it to a former state. Another suggested possibility would play with magnetic properties, with one hand used to attract virtual objects while the other pushes them away.